Method for devices in a network to participate in an end-to-end measurement of latency

ABSTRACT

A method of determining the latency of path segments in a communication network that uses multi-bit data packets comprises generating a test packet for use in determining the latency of path segments in the network; transmitting the test packet from a first device coupled to the network; storing in the test packet the time when a preselected bit in the test packet is transmitted from the first device; when the test packet is received by a second device coupled to the network, storing in the second device at least one of (a) the time when a preselected bit in the test packet is received by the second device and (b) the difference between (i) the time when the preselected bit in the test packet is transmitted from the first device and (ii) the time when the test packet is received by the second device.

This application is a continuation of and claims priority to U.S. patentapplication Ser. No. 13/542,449, filed Jul. 5, 2012, now allowed, whichis hereby incorporated by reference herein in its entirety.

FIELD OF THE INVENTION

This invention is directed towards creating a method for devices in anetwork to participate in an end-to-end measurement of latency and alsodetermine segment by segment latency without additional messaging in thenetwork.

BACKGROUND OF THE INVENTION

When an Ethernet circuit (or other type of circuit) is activated in anetwork, there is a need to be able to obtain precise performancemeasurements to make sure the circuit is fully functional in accordancewith the performance specification of the operator. Unidirectional(1-way) and bi-directional (2-way) delay measurements are an essentialperformance measurement that needs to be obtained as part of the serviceactivation. These measurements are also very useful to measure theperformance of the Ethernet circuit while IN SERVICE.

These measurements, though useful, do not take into account the multiplesegments that may exist within a network path and give no information toisolate the segment delay within the absolute path. To find such asegment over a multi-segment path requires numerous tests, excessivemessaging and time.

There is a need to be able to discover a segment by segment latencyalong a path when an end-to-end latency measurement is requested and notincrease messaging as a by-product of this segment by segmentmeasurement.

An example of a unidirectional (1-way) measurement is illustrated inFIG. 1. When a unidirectional delay measurement is requested betweennetwork devices 101 and 102 along a network path 103, a test packet 104is created and a timestamp 105 is inserted into the packet 104 denotingthe time when the first bit of packet 104 is transmitted. When thepacket 104 arrives at the network device 102, a second timestamp 106 istaken to denote the time when the last bit of the packet 104 arrives atthe device 102. The difference in time between timestamp 105 andtimestamp 106 denotes the delay in time to traverse the entire pathbetween devices 101 and 102. All intermediate nodes between the devicesare not deemed relevant. It is also to be noted that the clocks betweenthe network devices must be precisely synchronized by one of the manymethods known to one skilled in the art.

An example of a bi-directional (2-way) measurement is illustrated inFIG. 2. When a bi-directional delay measurement is requested betweennetwork devices 201 and 202 along a network path 203, a test packet 204is created and a timestamp 205 is inserted into the packet 204 denotingthe time when the first bit of the packet 204 is transmitted. When thepacket 204 arrives at network device 202, a second timestamp 206 istaken to denote the time when the last bit of the packet 204 arrives atdevice 202. Then the addresses of the test packet 204 are modified andtransmitted back to the device 201 containing the original timestamp205, with the timestamp 206 and a timestamp 207 that denotes with thefirst bit of the packet 208 is transmitted on the network. When thefinal bit of the packet 208 arrives at the network device 201, a finaltimestamp 209 is taken. The difference in time between timestamp 205 and206 ADDED to the difference in time between the timestamp 207 and 209gives the total round-trip delay of the bi-directional path. Anothermethod to determine the round-trip delay is to deduct the timestamp 205from the timestamp 209. All intermediate nodes between the devices arenot deemed relevant. It is also to be noted that the clocks between thenetwork devices must be precisely synchronized by one of the manymethods known to one skilled in the art.

SUMMARY OF THE INVENTION

In accordance with one embodiment, a method of determining the latencyof path segments in a communication network that uses multi-bit datapackets comprises generating a test packet for use in determining thelatency of path segments in the network; transmitting the test packetfrom a first device coupled to the network; storing in the test packetthe time when a preselected bit in the test packet is transmitted fromthe first device; when the test packet is received by a second devicecoupled to the network, storing in the second device at least one of (a)the time when a preselected bit in the test packet is received by thesecond device and (b) the difference between (i) the time when thepreselected bit in the test packet is transmitted from the first deviceand (ii) the time when the test packet is received by the second device.In one implementation, the time when a preselected bit in the testpacket is received by the second device is stored in the second device,and the latency of the path segment between the first and second devicesis determined to be the difference between the two stored times. Inanother implementation, the difference between (i) the time when thepreselected bit in the test packet is transmitted from the first deviceand (ii) the time when the test packet is received by the second device,is stored in the second device, and the latency of the path segmentbetween the first and second devices is determined by retrieving thedifference from the second device.

The test packet may be transmitted serially from the second device to aplurality of additional devices coupled to the network. Each time thetest packet is received by one of the additional devices, theinformation stored in the additional device includes at least one of (a)the time when a preselected bit in the test packet is received by theadditional device and (b) the difference between (i) the time when thepreselected bit in the test packet is transmitted from the first deviceand (ii) the time when the test packet is received by the additionaldevice. The test packet may also be returned from the second device tothe first device, or from one of the additional devices to the firstdevice via the same devices traversed by the test packet duringtransmission from the first device to the one additional device.

The test packet may be transmitted to and from the various devices whilenormal packets are being transported through the network.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention may best be understood by reference to the followingdescription taken in conjunction with the accompanying drawings.

FIG. 1 is a diagrammatic illustration of a one-way latency measurementin a packet-based communication network.

FIG. 2 is a diagrammatic illustration of a two-way latency measurementin a packet-based communication network.

FIG. 3 is a diagrammatic illustration of one-way latency measurementsfor path segments, as well as the entire path, in a packet-basedcommunication network.

FIG. 3 a is a diagrammatic illustration of modified one-way latencymeasurements for path segments, as well as the entire path, in apacket-based communication network.

FIG. 4 is a diagrammatic illustration of two-way latency measurementsfor path segments, as well as the entire path, in a packet-basedcommunication network.

FIG. 4 a is a diagrammatic illustration of modified two-way latencymeasurements for path segments, as well as the entire path, in apacket-based communication network.

DETAILED DESCRIPTION OF ILLUSTRATED EMBODIMENTS

Although the invention will be described in connection with certainpreferred embodiments, it will be understood that the invention is notlimited to those particular embodiments. On the contrary, the inventionis intended to cover all alternatives, modifications, and equivalentarrangements as may be included within the spirit and scope of theinvention as defined by the appended claims.

To allow the tracking of the delays of the individual segments of theend-to-end path of the circuit, a new field is defined within the packetthat is used during the test. This new value stores the time stampcreated when the first bit of the packet is transmitted for eachsegmented hop along the end-to-end path. This new value can be used byeach device along the path to calculate the delay from the last deviceto itself, while preserving the information needed to determine thedelay in the end-to-end path. This also allows devices that are notaware of this capability to operate normally. The intermediate hopdevices can then be interrogated later to easily find the segment delayproblem if the end-to-end path has an unacceptable delay measurement.

FIG. 3 illustrates an example of a unidirectional (1-way) measurement.When a unidirectional delay measurement is requested between a pair ofnetwork devices 301 and 302 along a network path 303 that includes anintermediate network device 304, a test packet 305 is created andtransmitted from the device 301 onto the path 303. A timestamp 306 isinserted into the packet 305 to denote the time when the first bit ofthe packet 305 is transmitted. A second timestamp field 307 is set atthe same timestamp value, as this is the originating device.

The first network device to receive the packet 305 along the path 303 isthe intermediate device 304. When the test packet 305 arrives at thedevice 304, the time difference ΔT1 between the time when the last bitis received and the value in the timestamp 307 is calculated and storedin the network device 304. (Alternatively, the timestamp can be takenupon receiving the first bit of the packet 305.) This value ΔT1 is thedelay of the path segment from device 301 to device 304. A new timestamp308, denoting the time when the first bit of the packet 305 istransmitted from the device 304, is taken and stored in the field wherethe timestamp 307 had been stored.

When the packet 305 arrives at the end-point device 302, the time whenthe last bit arrives is recorded in the device 302, and the timedifference ΔT2 between the timestamp 308 and the recorded time can bestored in the device 302 as the delay of the path segment from device304 to device 302. The time difference between the arrival time storedin device 302 and the time stored in 306 is the value of the end-to-enddelay in the path 303.

FIG. 3 a illustrates a modified embodiment of a unidirectional delaymeasurement made between network devices 301 and 302 along a networkpath 303 that includes two intermediate network devices 304 a and 304 b.A test packet 305 is created and transmitted from the first device 301onto the path 303, and a timestamp 306 is inserted into the packet 305denoting the time when the first bit of the packet is transmitted fromthe first device 301.

When the packet 305 arrives at the first intermediate device 304 a, thetime difference ΔT1 between the time when the last bit is received bythe device 304 a and the time when the first bit was transmitted fromthe device 301 (the value in timestamp 306) is calculated and stored inthe device 304 a. This value ΔT1 is the delay of the path segment fromdevice 301 to device 304 a. The test packet 305 is then forwarded to thenext network device 304 b in the test path.

When the packet 305 arrives at the third device 304 b, the timedifference ΔT2 between the time when the last bit is received by thedevice 304 b and the time when the first bit was transmitted from thedevice 301 (the value in timestamp 306) is calculated and stored in thedevice 304 b. Alternatively, the timestamp can be taken upon receivingthe first bit of the packet 305. The value ΔT2 is the total delay of thepath segments from device 301 to device 304 b. The test packet 305 isthen forwarded to the last network device 302 in the test path.

When the packet 305 arrives at the final device 302, the time differenceΔT3 between the time when the last bit is received by the device 302 andthe time when the first bit was transmitted from the device 301 (thevalue in timestamp 306) is calculated and stored in the device 302. Thisvalue is the total end-to-end delay of the path from device 301 todevice 302. The stored time differences ΔT1, ΔT2 and ΔT3 in therespective network devices 304 a, 304 b and 302 along the test path 303can then be retrieved centrally by any of several well known techniquesfor retrieving data from distributed network devices.

In FIG. 4, when a bi-directional delay measurement is requested betweennetwork devices 401 and 402 along a network path 403, a test packet 405is created and transmitted from the device 401 along the path 403. Atimestamp 415 is inserted into the packet 405, denoting the time whenthe first bit of the packet 405 is transmitted. A second timestamp field406 is set at the same value as the timestamp 415, as this is theoriginating device.

When the packet 405 arrives at an intermediate network device 404, thetime difference ΔT1 between the time when the last bit is received bythe device 404 and the value in the timestamp 406 is calculated andstored in the device 404. This value ΔT1 is the delay of the pathsegment from device 401 to device 404. A new timestamp 407, denoting thetime when the first bit of the packet 405 is transmitted from the device404, replaces the timestamp 406 previously stored in the packet 405.

When the packet 405 arrives at the third device 402, the time when thelast bit in the packet 405 arrives at the device 402 is recorded in thedevice 402 as a timestamp 408. The time difference ΔT2 between timestamp408 and the time stored in timestamp 407 is calculated and stored in thedevice 402. This value ΔT2 is the delay of the path segment from device404 to device 402.

Next, the addresses of the original test packet 405 are reversed in apacket 410 that is transmitted in the reverse direction along thenetwork path 403, from device 402 to device 401 via the intermediatedevice 404. The packet 410 still contains the original timestamp 415,the recorded timestamp 408 and new timestamps 409 and 411, both denotingthe time when the first bit of the packet 410 is transmitted from thedevice 402.

When the packet 410 arrives at the intermediate network device 404, thetime difference ΔT3 between the time when the last bit is received bythe device 404 and the value in the timestamp 409 is calculated andstored in the device 404. This value ΔT3 is the delay of the pathsegment between from device 402 to device 404. A new timestamp 414,denoting the time when the first bit of the packet 415 is transmittedfrom the device 404 toward the device 401, replaces the timestamp 409 inthe packet 410.

When the packet 410 arrives at the third device 401, which is the endpoint of the return path, the time when the last bit in the packet 410arrives at the device 401 is recorded in the device 401 as a timestamp416. The time difference ΔT4 between the recorded time 416 and the timestored in timestamp 414 is calculated and stored in the device 401. Thisvalue ΔT4 is the delay of the path segment between from device 404 todevice 401.

Also, the difference in time between timestamp 415 and 408 ADDED to thedifference in time between timestamp 411 and 416 gives the totalround-trip delay of the bi-directional path. An alternative method tocalculate the total round-trip delay is to deduct the timestamp 415 fromtimestamp 416. It is also to be noted that the clocks between thenetwork devices must be precisely synchronized by one of the manymethods known to one skilled in the art.

In another embodiment illustrated in FIG. 4 a, a two-way delaymeasurement is made between network devices 401 and 402 along a networkpath 403 that includes an intermediate network device 404. A test packet405 is created at the first device 401, and a timestamp 415 is insertedinto the packet 405 denoting the time when the first bit of the packetis transmitted from the first device 401. When the packet 405 arrives atthe second device 404, the time difference ΔT1 between the time when thelast bit is received by the device 404 and the time when the first bitwas transmitted from the device 401 (the value in timestamp 415) iscalculated and stored in the device 404. This value ΔT1 is the segmentdelay of the path segment from device 401 to device 404. The test packet405 is then forwarded to the next network device 402 in the test path.

When the packet 405 arrives at the third device 402, the time differenceΔT2 between the time when the last bit is received by the device 402(stored as timestamp 408) and the time when the first bit wastransmitted from the device 401 (the value in timestamp 415) iscalculated and stored in the device 402. The value ΔT2 is the totalsegment delay of the path segment from device 401 to device 402.

Then the addresses of the original test packet 405 are reversed in apacket 410 that is transmitted to the device 401 in the reversedirection along the network path 403. A timestamp 409 is inserted intothe packet 410, denoting the time when the first bit of the packet istransmitted from the device 402 onto the return path. Timestamps 415 and408 are also inserted into the packet. When the packet 410 arrives atthe intermediate device 404, the time difference ΔT3 between the timewhen the last bit is received by the device 404 and the time when thefirst bit was transmitted from the device 402 (the value in timestamp409) is calculated and stored in the device 404. This value ΔT3 is thedelay of the path segment from device 402 to device 404. The test packet410 is then forwarded to the network device 401, which is the end pointof the path being tested.

When the packet 410 arrives at the device 401, the time difference ΔT4between the time when the last bit is received by the device 401 (storedas timestamp 416 in device 401) and the time when the first bit wastransmitted from the device 402 (the value in timestamp 409) iscalculated and stored in the device 401. This value ΔT4 is the delay ofthe path from device 402 to device 401.

The stored time difference values ΔT1, ΔT2, ΔT3, and ΔT4 stored in thenetwork devices 401, 402 and 404 along the test path can be retrievedcentrally. It will be evident to those skilled in the art that theinvention is not limited to the details of the foregoing illustratedembodiments and that the present invention may be embodied in otherspecific forms without departing from the spirit or essential attributesthereof. The present embodiments are therefore to be considered in allrespects as illustrative and not restrictive, the scope of the inventionbeing indicated by the appended claims rather than by the foregoingdescription, and all changes which come within the meaning and range ofequivalency of the claims are therefore intended to be embraced therein.

Also, the difference in time between timestamp 415 and 408 ADDED to thedifference in time between timestamp 409 and 416 gives the totalround-trip delay of the bi-directional path. An alternative method tocalculate the total round-trip delay is to deduct the timestamp 415 fromtimestamp 416. It is also to be noted that the clocks between thenetwork devices must be precisely synchronized by one of the manymethods known to one skilled in the art.

The invention claimed is:
 1. A method of determining a round triplatency of a network path in a communication network having at least twopath segments, that uses multi-bit data packets, said method comprising;generating a test packet for use in determining the latency of each ofsaid at least 2 path segments in said network path; transmitting saidtest packet between first and second devices coupled to said network atopposite ends of a first path segment, and then between said seconddevice and a third device coupled to opposite ends of a second pathsegment; and returning said test packet to said first device bytransmitting said test packet between said third and second devices andthen between said second and first device; storing in said test packet atransmit time when a first preselected bit in said test packet istransmitted from said first device; when said test packet is received bysaid second device coupled to said network, storing in said seconddevice the difference between (i) the transmit time when said firstpreselected bit in said test packet is transmitted from said firstdevice and (ii) a receipt time when a second preselected bit in saidtest packet is received by said second device; and when said test packetis received by said third device coupled to said network, storing insaid third device the difference between (i) the transmit time when saidpreselected bit in said test packet is transmitted from said firstdevice and (ii) a receipt time when a second preselected bit in saidtest packet is received by said third device; and determining the roundtrip latency of the network path when said test packet is returned fromthe third device to said first device via the same devices traversed bysaid test packet during transmission from said first device to saidthird device.
 2. The method of claim 1 in which said receipt time is thetime when the second preselected bit is received by said second device,and which includes determining the latency of the path segment betweensaid first and second devices by determining the difference between thetwo stored times.
 3. The method of claim 1 in which said differencebetween (i) the transmit time when said first preselected bit in saidtest packet is transmitted from said first device and (ii) the receipttime when the second preselected bit in said test packet is received bysaid second device wherein said receipt time is stored in said seconddevice, and includes determining the latency of the path segment betweensaid first and second devices by retrieving said difference from saidsecond device.
 4. The method of claim 1 in which said test packet istransmitted serially from said second device to a plurality ofadditional devices coupled to said network; and each time said testpacket is received by one of said additional devices, storing in saidadditional device at least one of (a) a receipt time when a preselectedbit in said test packet is received by said additional device and (b)the difference between (i) the transmit time when said preselected bitin said test packet is transmitted from said first device and (ii) thereceipt time when said preselect bit in said test packet is received bysaid additional device.
 5. The method of claim 1 in which said testpacket is returned from said second device to said first device.
 6. Themethod of claim 1 which includes; storing in said test packet a transmittime when a preselected bit in said test packet is transmitted from saidsecond device, when said test packet is received by a third devicecoupled to said network, storing in said third device a receipt timewhen said second preselected bit in said test packet is received by saidthird device, and determining the latency of the path segment betweensaid second and third devices by determining the difference between thesecond-device transmit time stored in said test packet and the receipttime stored in said third device.
 7. The method of claim 6 in which saidtest packet has first and second timestamp fields, and which includesstoring in said first field the transmit time when a preselected bit insaid test packet is transmitted from said first device, and storing insaid second field the transmit time when a preselected bit in said testpacket is transmitted from said second device.
 8. The method of claim 1in which said test packet is transmitted from said first device to saidsecond device while normal packets are being transported through saidnetwork.
 9. An apparatus for determining a round trip latency of anetwork path in a communication network, having at least two pathsegments comprising a first and second path segment, each having a firstand second end, wherein said communication network uses multi-bit datapackets, comprising: a first network device at the first end of thefirst path segment being adapted to generate a test packet and transmitthe test packet; a second network device at the second end of the firstpath segment and at the first end of the second path segment beingadapted to receive the test packet, to store a first packet send time ofthe test packet from the first device to the second device, to use thestored first packet send time to calculate a latency of a first pathsegment between the first device and the second device, and to transmitthe test packet; a third network device at the second end of the secondpath segment, said third network device adapted to receive the testpacket, to store a second packet send time of the test packet from thesecond device to the third device, to use the second packet send time tocalculate a latency of a second path segment between the second deviceand the third device, and to transmit the test packet back to the secondnetwork device; the second network device being adapted to receive thetest packet from the third network device, to store a third packet sendtime of the test packet from the third device to the second device, touse the stored third packet send time to calculate a latency of thesecond path segment between the third and second device, and to transmitthe test packet back to the first network device; the first networkdevice being adapted to receive the test packet from the second networkdevice, to store a fourth packet send time of the test packet from thesecond device to the first device, and to use the fourth packet sendtime to calculate a latency of the first path segment between the seconddevice and the first device; the third network device being adapted todetermine the latency of the network path in a first direction bycombining the latency of the first and second path segments; and thefirst network device being adapted to determine the round trip latencyof the network path by combining the latency in the first direction andthe latency of the first and second path segments when said test packetis returned from the third device to said first device via the samedevices traversed by said test packet during transmission in the firstdirection.
 10. The apparatus of claim 9 wherein the first packet sendtime is a time difference between the time a preselected bit is receivedby the second device and a time when a preselected bit is transmittedfrom the first device.
 11. The apparatus of claim 9 wherein the secondpacket send time is a time difference between the time a firstpreselected bit is received by the third device and the time a secondpreselected bit is transmitted from the second device.
 12. The apparatusof claim 9 wherein the third packet send time is a time differencebetween the time a first preselected bit is received by the seconddevice and the time a second preselected bit is transmitted from thethird device.
 13. The apparatus of claim 9 wherein the fourth packetsend time is a time difference between the time a first preselected byis received by the first device and the time a second preselected bit istransmitted from the second device.
 14. The apparatus of claim 9 inwhich said test packet is transmitted serially to a plurality ofadditional devices coupled to said network.
 15. The apparatus of claim14 wherein a packet send time of the test packet is stored in each ofsaid plurality of additional devices and the latency of each of the pathsegments is determined by the stored send times between each of saidplurality of additional device.
 16. The apparatus of claim 15 whereinthe round trip latency of the network path is determined by adding thelatency of the path segments when said test packet is returned to saidfirst device via the same devices traversed by said test packet duringtransmission from said first device to the last of the plurality ofadditional device.